{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization\n",
    "\n",
    "Regularization is the technique used to solve the overfitting problem. An overfitted model means that the model can predict very well with the training data, but perform poorly with independent validation data.\n",
    "\n",
    "When we add more predictors to our model, we will almost neccessarily decrease the **Residual Sum of Squares** (RSS; smaller RSS indicates better model). This increases the complexity of our model and makes our model only perform well on the training data (overfitting).\n",
    "\n",
    "To balance the RSS and model overfitting, we introduce penalty for adding new predictors (coefficient $\\beta \\neq 0$) to the model.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## LASSO regularization and Ridge regularization\n",
    "\n",
    "* **LASSO**: $min \\{RSS + \\lambda\\sum_{j=1}^{p}|\\beta_1|\\}$\n",
    "* **Ridge**: $min \\{RSS + \\lambda\\sum_{j=1}^{p}\\beta^2_2\\}$\n",
    "\n",
    "## Elastic Net regularization\n",
    "\n",
    "Elastic net is a regularized method that linearly combines penalities of the lasso and ridge methods.\n",
    "\n",
    "* **elastic net**: $min \\{RSS + \\lambda[\\sum_{j=1}^{p}\\frac{1}{2}(1-\\alpha)|\\beta^2_2| + \\alpha\\sum_{j=1}^{p}\\beta_1]\\}$\n",
    "\n",
    "Reference: https://spark.apache.org/docs/2.1.1/ml-classification-regression.html\n",
    "\n",
    "## *regParam* and *elasticNetParam* parameters in regression models\n",
    "\n",
    "* **regParam**: corresponds to $\\lambda$\n",
    "* **elasticNetParam** corresponds to $\\alpha$. When $\\alpha = 0$, it is ridge regularization (L2 penalty). When $\\alpha = 1$, it is lasso regularization (L1 penalty)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
